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Introduction 


• Study Overview 

- Understand the primary reasons for cost and schedule growth 

- Determine percentage of growth outside of the project’s control 

• Approach 

- Examined project documentation, including CADRe, milestone presentations, 
monthly project reports, etc., to develop a case history of each project 

- Conduct interviews with key project personnel to provide the insight required to 
understand all reasons for growth 

- Allocated growth events to “Explanation of Change” EoC “tree” to quantify 
reasons for growth 

• Timeline 

- First set of 10 NASA historical missions reviewed November/December 2009 

- Results of second set of 10 missions & 20 mission total reviewed in this briefing 

- Remaining set of missions, analysis & recommendations completed by October 





Phase II Executive Summary 


• Mission data collected for twenty NASA science missions 

• Binning process used to identify cause of cost growth 

• Results of Mission Explanation of Change (EoC) categorization 

- 5% External to NASA 

- 22% External to the Project 
-34% Relative to Project Planning 

- 39% Relative to Project Execution 

• Following Aggregate Cost Increase, Not Including Reserves, Observed 

-19% additional cost increase due to HQ and external factors 

-24% cost increase from Phase B start until realistic programmatic baseline 
established typically after confirmation 

- 27% cost increase after realistic programmatic baseline established 


Interview Preparations 


• Case histories developed 

- NASA science mission cases developed including references 


• Assess initial EoC Categorization 

- Initial allocations developed while waiting for interviews 

• Standard template developed to “standardize” EoC categorization 


• For each mission, preparation of case summary provided to 
interviewees 


- Key event timeline 

- Mission Cost growth bar chart 

- WBS element cost growth pie chart 

- Schedule summary chart 

- Reasons for cost growth 

- Additional questions 


erviews provided input to modify initial EoC Categorization 


Interview Approach 


• Speak to broad range of project personnel if available 

- Project Manager 

- Spacecraft Manager 

- Payload Manager 

- Project Systems Engineer 

- Resource Manager 

• Review organization charts from milestone documents to determine 
most appropriate personnel 

- Tried to identify personnel who were in place when primary cost growth 
occurred (typically CDR) 

- Recommended to Center POCs specific personnel to interview 

- Center POCs checked for availability of personnel 

- Typically, a subset of project personnel were available for interview 

Interviews were required to assess undocumented development issues 


Common Themes Discovered Only By Interview Process 


• Optimism in Initial Proposed Design & Scope 

- Underscoping of effort 

- Expectations of subcontractor 

- Pressure from HQ or Center to reduce cost for Directed missions 

• Overselection of Competed Instruments on Directed Missions 

- Competitive Instrument AO process led to instruments being selected that: 

• Were outside the technical constraints, or 

• Had very little chance of meeting technical constraints, or 

• Had greater number of instruments than originally planned 

• Organizational Issues 

- Initial roles and responsibilities were not clearly defined and therefore 
expectations were different leading to discontent and mistrust among 
different organizations 

Interviews were invaluable in providing insight into undocumented issues in development 





Cost Growth Summary - Combined 20 Mission Average 
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Note: Development cost represents Phase B/C/D cost including reserves but not including launch vehicle cost 
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Comparison of Competed vs. Directed Missions 


Development Cost Growth 


50% -| 4 5% 4 5 % 
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Development Schedule Growth 



From Phase B From PDR From CDR 
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Spacecraft Dry Mass Growth 



From Phase B From PDR From CDR 
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Payload Mass Growth 



From Phase B From PDR From CDR 

Start 


For Missions in study, Competed Missions have similar cost & schedule growth 









Development Cost Growth from PDR 


Interrelationship of Cost & Schedule Growth 
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Schedule Growth from PDR 

Cost will typically grows as Schedule grows 


Earth Orbiting Missions 

% Dev. Cost Growth = 0.7775 * % Sched. Growth + 19.4% 

R 2 = 0.5503 
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Planetary Missions 
\% Dev. Cost Growth = 0.2733 * % Sched. Growth + 14%; 

R 2 = 0.4174 



Explanation of Change (EoC) Categorization “Tree” Example 


• Standard worksheet allows display/calculation of roll-up results 







EoC Event List Example 


• Events, quantification and rationale are documented in worksheet 

• Events are allocated to different elements in EoC tree 



Total 

$ 

66.0 Should match total growth for Project 

Explanation of Change Categorization 

Code 

Value ($; Rationale 

j RP2-Project Pianning-Design-Instrument 

3 p P2 

s 

7.8 PABSI descoped but cost increase because CPR and HVPS designs were underscoped - The CPR stru 

j PP1 -Project Pla nning-Design-Spacecraft 

kjppi 

$ 

7,4 SC bus design change - Changes include shortening of the bus to fit in DPAF envelop and change to ac< 

| PP5-Project Planning-Programmatic 

3pF5 

$ 

2,6 Underscoping of mission assurance and SE effort 

j PP7- Project Planning-Other 

3 ppj 

1 $ 

1.8 Mission design change to formation fly with EOS-Aqua requiring additional reviews and change in MSRD 

j HQ2-External to the Project-Agency Level (HQ)-Program Requirements 

2 hq6 

$ 

5.1 : March 2003 to April 2004 launch delay due to adding Cal ipso co-manifest - estimated as proposed cost 

] PE4-Project Execution-System Oevelopment-Instrument 

g P E 4 

: $ 

9>2 CPR delivery delay slowing down the SC team and leading to bath tub periods - i&T problems such as tf 

| PE3 -Project Execution-System Development- Spacecraft 

g| p £3 

$ 

5.7 Typical development issues associated with spacecraft development 

| PE6- Project Execution-System Development-Ground Systems 

wJ PE6 

S 

3.2 Typical development growth 

PE4- Project Execution-System Development-Instrument 

[jj p E4 

S 

1.2 PM/SE/MA Growth allocated to difficulties with instrument development 

PP2 -Project Planning-Design-Instrument 

Q pp2 

$ 

2.1 CPR transmitter being picked up by JPL since CSA could not make it with their proposed budget 

HQ2-External to the Project-Agency Level (HQ)-Program Requirements 

S hq2 

: $ 

' " " 'X ' ••"••••• ■ ' ' ' ' 

4.0 Additional requirements imposed to project (ex.. T 7120,x and 7119.x), additional reviews, ITAR 

j HQ2-External to the Project-Agency Level (HQ)-Program Requirements 

S hq2 

$ 

2.7 Launch slip from April 2005 to June 2005 due to readiness of CALIPSO and also NOAA’s N launch dela} 

j None 

1 w 1 



| NE1-NASA External-Launch Vehicle 

Qnei 

$ 

1.4 Instrument cost due to 11 month LV delay 

| NE1-NASA External-Launch Vehicle 

Q nei 

$ 

5.2 Spacecraft cost due to 1 1 month LV delay 

) NE1-NASA External-Launch Vehicle 

0NE1 

$ 

2.8 PM/SE/MA cost due to It month LV delay 

j NE1-NASA External-Launch Vehicle 

kjNEI 

: $ 

3.8 GDS/MOS/Science cost due to 11 month LV delay 




Summary of Findings: Allocated Cost Increase for New 
Missions 


• Growth Internal to the Project due to Execution (39%) 

- Both spacecraft and instrument issues occurred 

- Difficulty in objectively assigning instrument vs. spacecraft may not show 
true impact 

• Growth Internal to the Project due to Planning (34%) 

- Initial optimism seems to have played a large part 

- Typically allocated prior to establishing “realistic” programmatic baseline 

• Growth External to the Project (22%) 

- Primary issues were additional HQ or program requirements like NIAT 

• Growth External to NASA (5%) 

- Primarily Industrial base issues like LV delays 


Initial Data Set Can Be Used to Develop Reserve Level 
Sanity Check 

• Based on cost increases from each broad category, a “required 
reserve” percentage can be calculated to determine the additional cost 
required to cover increase experienced 

• These initial observations, however, are based on this small set of 20 
NASA Science missions and may change significantly if Human Flight 
or larger scale missions are added 

• These observations can also provide a focus area for either reducing 
perceived cost growth or reducing the cost of missions as a whole 

- Current approach suggests adding appropriate buffer to cover natural 
design evolution and typical challenges to reduce perceived cost growth 

- Greater emphasis on more robust initial estimates (i.e. JCL) may reduce the 
“collateral damage” cost by fitting more clearly within initial budget caps 


Agenda 


• Study Overview 

- Task Overview 

• Progress 

- Case Studies 

- Interviews 

• Pre-Interview Results Summary 

- Cost & Schedule Growth 

- EoC Categorization Example/Summary 


Next Steps 


Potential Science Mission Selections for Phase III 


Mission List 


Number 

Round 

Missions 

Center 

21 

3 

Cassini 

JPL 

22 

3 

Chandra 

Other 

23 

3 

GALEX 

JPL 

24 

3 

RHESSI 

GSFC 

25 

3 

EO-1 

GSFC 

26 

3 

Landsat-7 

GSFC 

27 

3 

TRMM 

GSFC 

28 

3 

WISE 

JPL 

29 

3 

SDO 

GSFC 

30 

3 

TIMED 

APL 

31 

3 

NEAR 

APL 

32 

3 

THEMIS 

Other 

33 

3 

Stardust 

JPL 


Distribution 


Theme 


Planetary 

Center 

Count 

Distribution 

Astro 

JPL 

14 

42% 

Astro 

GSFC 

12 

36% 

Astro 

APL 

4 

12% 

Earth 

Other 

3 

9% 

Earth 

Total 

33 


Earth 

Astro 

Theme 

Count 

Distribution 

Helio 

Earth 

7 

21% 

Helio 

Planetary 

12 

36% 

Planetary 

Astro 

8 

24% 

Helio 

Helio 

6 

18% 

Planetary 

Total 

33 
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Example of Recommendations - Potential Changes to AO 
Selection Process 

• To remove potential optimism in initial proposal submittal, behavior must be changed 

- Incentive to maximize science irrespective of cost risk must be removed 


• Example of Acquisition Policy Change 

- Current: Choose maximum science at similar perceived cost 

• Assuming all proposed cost are at cap and not unachievable (i.e. not high cost risk), pick the 
proposal that has the greatest science value 

• Example: Kepler & Dawn selection 

• Requires more robust estimating to anticipate growth to minimize threat of breach 



Potential Change: Choose lowest cost risk for equal science 

• Once Step 1 downselect occurs, all science deemed Category 1 and considered equal, 
allowing the proposal with the lowest cost risk to be selected 

• Example: GRAIL selection premise 

• Requires “advertising” in AO solicitation combined with repeated demonstration by 
selections providing message to community that change is permanent and real 

• Should increase chance of mission staying within cost cap thereby minimizing chance of 
collateral damage 
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Summary 

• Mission data collected for twenty NASA missions 

- Detailed data collected and referenced 

- Data summarized and assessed 

• Interviews conducted and were found to be extremely valuable 

• Results of 20 Mission Explanation of Change (EoC) categorization 

-5% External to NASA 

- 22% External to the Project 
-34% Relative to Project Planning 
-39%o Relative to Project Execution 

• Next steps 

- Additional NASA science missions plus pursuing ISS elements 

- Formulating recommendations based on interviews and supporting data 

- Final mission results & recommendations presented October 2010 





